Methods and apparatus for performing media/device sensitive processing of messages stored in unified multimedia and plain text mailboxes

ABSTRACT

The invention provides users of either a unified messaging mailbox or simple plain text storage mailbox, with media and device sensitive methods and apparatus to convert messages from one medium to another (when necessary) taking into account the type of target device (for example, cell phone, PDA, pager, etc.), being used to access a stored message Additionally, the invention provides methods and apparatus for determining target device type and for summarizing message content based on target device type.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to methods and apparatus for performingmedia/device sensitive processing of messages stored in unifiedmultimedia and plain text mailboxes. More particularly, the inventionrelates to methods and apparatus for (1) converting messages from onemedium to another (when necessary); taking into account the type oftarget device (for example, cell phone, PDA, pager, etc.), being used toaccess a stored message; (2) determining target device type; and (3)summarizing message content based on target device type.

2. Brief Description of the Prior Art

Business people receive many different kinds of messages, e.g.electronic mail, voice mail, fax, video messages, attachments toelectronic mail. It is possible and desirable to have all messages sentto a single mail box from which they may all be retrieved regardless ofthe message type. However, the only retrieval device which is capable ofreading all of these different types of messages is a personal computerhaving a graphical display and audio video capability. Unfortunately, itis not always possible or convenient to retrieve messages with apersonal computer.

A unified mailbox where all kinds of media (voice, fax, e-mail, andvideo) are made accessible and/or visible from virtually anywhere to asubscriber or user in one basket is a convenient means of communicationwhen compared to handling multiple mailboxes with distinct media.Current solutions for a unified mailbox are inefficient, however, forsomeone with an intense communication style and a frequent need tohandle his/her messages remotely. The mismatch of media type of theinformation and the capabilities of the various (often limited) devicesused for remote access places a heavy burden on the user and theinterface of the system. This is especially true for the interfacesutilizing a telephone with no display, or handheld devices with limiteddisplay capabilities.

Some of the problems arise in the context of compound and/or lengthymessages in connection with one or the other access means. For example,it is not possible to deliver voice and fax messages to a text-onlye-mail capable device. It is also difficult to deal with lengthy e-mailsdelivered to a voice-only interface or to a text-interface with limitedcapabilities. Even when the device has a fully functional GUI interface,there is room for increased efficiency with large amounts of data. It isa challenge to efficiently present the information in various officedocument formats (e.g., Word Processor, Spreadsheet, and Presentations)associated with a message. It is often difficult to locate and visuallypresent related messages and attachments. When the mailbox has manymessages in it, it is difficult to reference the messages.

Other problems arise due to the increased amount of information theunified mailbox can provide. Current mechanisms for organizing andpresenting relationships among messages (listing by arrival time,subject, sender, etc.) are insufficient for a large number of messagesof varying media and, especially, mixed media within a given message.

It would be desirable to provide a flexible, media independent way offinding and navigating related messages. With current systems, forexample, the user is unable to recognize that there is a relationshipbetween a voice message and a fax without listening to the voice messageand displaying/printing the fax.

Because the presentation of unified mailbox information is more complex,especially if relationships as described hereinabove are incorporatedinto the presentation, identifying an individual item (message ormessage attachment) for further action can become problematic. How doesthe client/user identify to the server which message is to be actedupon? Are the entire message and its attachments to be involved? Is it asingle attachment or only the original message body? And if the messagesare presented in a “graph” format, how does the user select anindividual item?

Current unified mailbox systems offer media sensitivity for messageretrieval only when accessed with a graphical user interface (GUI) froma PC client. If a particular media or office document is attached to ane-mail, the user needs to click-on it in order to launch a specificapplication, for example, an audio player for voice, tiff-viewer forfax, video player to view a video message, etc.

For users with intense communication requirements (e.g. executives orcustomer service agents who receive hundreds of compound messages daily)there are no means to quickly process inbox messages except by thesender information, the subject line, and maybe few lines of the messagebody. In order to read messages, the user has to click on or mark acertain item in a graphical interface in order to get to the messagebody.

No content summarization of lengthy text messages or respectiveattachments is available yet that would remarkably improve theefficiency of handling the daily information avalanche in the office.

Current mailbox searching does not provide visual display of content andtemporal relationships. No search capability exists yet for non-textmessages.

If a unified mailbox is accessed from a telephone interface, voice ande-mail messages are retrievable and the user can listen to both.Existing text-to-speech technology provides a means to convert thee-mail to voice. A fax message can be forwarded to a fax machine orprinter.

However, if an e-mail contains an attachment, the systems are able toindicate that, but are unable to access its content. Similarly, thecontents of a fax or other documents attached to an e-mail are indicatedbut not accessible to the user accessing the mailbox with a telephoneinterface.

If an e-mail is lengthy, the user may be able to navigate through it byaccelerating the text-to-speech reading speed. However, there is nomeans of text content summarization applied to shorten the processwithout eventually losing/skipping critical content.

If messages are forwarded to a handheld device via a wireless servicebut the device has limited text-display capabilities only certain partsof the email (From, Subject and a limited number of characters of themessage body) can be displayed. If the critical information in themessage is not in the beginning of the message body that is displayed,it is “lost” to the recipient. He/she has to use other access methods ormake a call into the messaging system/server to retrieve the full textmessage (by listening to it or by initiating a printing to a devicenearby).

As mentioned above, voice and other media attachments are indicated butnot transmitted and/or displayed on a text-only display. The user needsto use other access methods to retrieve the messages. Additionally, notext content summarization methods are utilized to deal with accessdevice technology limitations.

Full message sensitivity is only provided when accessing a mailbox witha multimedia PC. However even multimedia PCs lack any means to summarizemessage content in order to make it more efficient for the recipient toread his/her lengthy messages. Also, there are yet no means to summarizecontent of attached documents.

When accessing a mailbox with a telephone, the media and devicesensitivity is limited to voice and e-mail. Again, no techniques of textcontent summarization are applied yet in order to make the retrieval ofthe message information over the phone more convenient.

In the case of handheld or mobile devices with limited text-displaycapabilities, the problem is that lengthy messages are usually nottransmitted in their entirety by the wireless/paging service providers.Additionally, any other media attachments are “lost”. No contentsummarization of lengthy text messages or respective attachments isavailable yet that would remarkably improve the efficiency of handlingthe daily information avalanche in the office.

SUMMARY OF THE INVENTION

It is therefore an object of the invention to provide methods andapparatus for accessing multimedia messages from a unified mailbox.

It is also an object of the invention to provide methods and apparatusfor converting media types in a unified multimedia mailbox.

It is another object of the invention to provide methods and apparatusfor summarizing the content of messages in a unified multimedia mailbox.

It is yet another object of the invention to provide methods andapparatus for cross referencing related messages based on content.

It is another object of the invention to provide methods and apparatusfor improved handling of email attachments.

It is still another object of the invention to provide methods andapparatus for customizing mail handling based on a system profileadapted to the device used to access the mailbox.

In accord with these objects which will be discussed in detail below theapparatus and associated methods of the invention include a mail serverthat provides multimedia message inbox for one or several users on anetwork; a subsystem that detects media attachments to messages in amailbox; a subsystem that converts media attachments into another mediatype using text-to-speech, fax-to-text, video voice track into text andspeech-to-text; a subsystem that analyzes and summarizes the textcontent of original or converted media in respect of the linguisticmeaning; a subsystem that delivers appropriate media according to anaccess device and message purpose, as defined in a profile; a subsystemthat identifies cross-media interrelationships between messages andcontrols the media conversions necessary for this analysis; and asubsystem that controls a reference number scheme.

The methods and apparatus of the invention solve the problems discussedabove by utilizing advanced media conversion methods, analysis andsummarization of message content, and intelligent forwarding concepts.It provides access device and media sensitive intelligence for a mailboxwhen retrieving or forwarding a particular message.

The concept of media conversion is extended beyond text-to-speech toother attachments; a speaker-independent, large vocabulary,telephony-quality speech recognition engine is utilized to convert avoice message to text or to convert the voice track of a videoattachment into readable text. Similarly, fax information is convertedinto text.

According to the invention, the content of messages is automaticallysummarized. The summarization of a message content is an improvementtoward efficiency, particularly in the case of a forwarded lengthymessage to a handheld device with limited display capabilities. The sameis true for reading a lengthy message over the phone. Summarization isalso applied to attached media (e.g. fax, Word document) extends eventhe media content accessible.

Both, the media conversion and the content summarization appliedtogether provide compatibility with the access device. Depending on theuser, the types of potential access devices are usually predefined;therefore messages along with their attachments that form the messagecontent can be tailored to those devices while accessed or forwardedaccording to a profile. This ensures the availability of moreinformation to the recipient at the device of choice and that isprobably most convenient. Still, if the user requires more information,he/she can utilize another access method.

The invention also provides cross-media searching and visual displaying.Often messages related to a specific topic of interest to the user arein different media and spread throughout the message store (e.g. indifferent folders). The cross-media search finds these messages andpresents them to the user in a way that makes the content and timerelationships clear allowing efficient use of the otherwise overwhelmingamount of information. The search engine utilizes sophisticatedlinguistically based analysis tools to discover the messagerelationships.

Additionally, a reference number scheme for all messages is provided.All messages in a particular group of messages of interest to the userare assigned a reference number to be used in further actions. Thus aPDA user can, for example, get a summary of messages with referencenumbers and an indication of the message type. This reference number maythen be used to access that message, and through it, a particularattachment to that message for further. Voice commands may be used toinvoke actions on items more efficiently using the reference numbers ofmessages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high level block diagram of a multimedia mail systemaccording to the invention.

DETAILED DESCRIPTION

Turning now to FIG. 1, an integrated multimedia messaging systemaccording to the invention includes a mail server 10 that providesmultimedia message inbox for one or several users on a network via oneor more target devices 9; a mail processor 11; a subsystem 12 thatdetects media attachments to messages in a mailbox; one or moresubsystems that converts media attachments into another media type usingtext-to-speech 14, fax-to-text 16, video voice track into text 18 andspeech-to-text 20; a subsystem 22 that analyzes and summarizes the textcontent of original or converted media based on, for example, keywordlists 23 in respect of the linguistic meaning; a subsystem 24 thatdelivers appropriate media according to an access device and messagepurpose, as defined in a profile; a subsystem 26 that identifiescross-media interrelationships between messages and controls the mediaconversions necessary for this analysis; a subsystem 27 that filtersmessages; and a subsystem 28 that controls a reference number scheme.

The invention can better be understood through an illustrative examplesuch as the notification of a single-media voice message to a datapager. The following describes an example of this process involving auser that has a multimedia mailbox and a data pager who receives a voicemessage.

The problem is to provide the “best” information to the pager so theuser can proceed most efficiently. What is the “best” information willvary according to the user's actual preferences, but will most likelyinclude sender identification and meaningful portions of the messageitself. In addition, there are probably messages the user would preferto delay any handling of until an appropriate device is available. Thusthe steps for sending voice messages to a pager would include: a)filtering messages to be processed, b) speech-to-text conversion, c)summarization and post filtering, and d) selection and delivery of textinformation to the device.

Since the resources involved in processing a message may be large,messages are pre-filtered. Speech-to-Text is “expensive” in its use ofresources. Interrupting the user with any but the most importantmessages can be an unnecessary expense of the user's time and attentionas well as a waste of system resources. Thus a mechanism to prevent thepresentation of a message to a given device is important. This filteringis based on a variety of data including sender, message priority, etc.and the criteria for filtering is stored in the system profile for theuser.

Voice messages which pass through the pre-filter are converted to text.This is most efficiently accomplished on the server side, perhaps with adedicated “helper” server explicitly for the server so as not to disturbother processing on the server. The resulting text message is then beassociated with the original message (as the text message body or as aseparate attachment).

Before sending the text message to the pager, it is subjected topost-conversion filtering and summarization. Post-conversion filteringis optional, preventing processing of messages that appear not to be ona topic deemed important to the user. If it does not appear important,it would then remain in the mailbox to be processed. If the messagesurvives the post-conversion filtering step, the text is thensummarized.

Most simply, summarization includes reduction to a list of keywords andphrases found within the text. The summarization is created by removingfrom the message words/phrases not found within the user-defined list ofkeywords/phrases. More complex summarization includes allowing the userto specify the keyword/phrase list based on the sender of the message.

Since the message is a speech-to-text conversion, the keywords and theirhomonyms should be checked. An option on the summarization, for examplea check box that says “allow homonyms”, could be utilized to enable thisfeature.

Even more complex summarization methods contemplated by the inventioninvolve performing sophisticated grammatical parsing and analysis.

Data is transmitted to the pager based on a user defined data selectioncriteria which is stored as a template in the system profile for theuser. The data available for selection includes sender same, time,summary, message priority, un-summarized text, and other fields asavailable.

The user describes a template that indicates the information desired andthe number of characters of each field desired. For example:

-   -   “From % SENDER % at % TIME %: %100SUMMARY % ”        indicates that the user wants a string that includes the entire        sender name, the received time and the first 100 characters of        the summary to appear on his pager.

When the user receives the page, the summary information gives him/herenough information to determine how critical the message is. If itappears critical, he/she may choose to access the entire message using adifferent device, e.g. a telephone.

Another example is the retrieval of text messages (such as email) via atelephone. Text messages are pre-filtered as described above. The textis then summarized. The summary is then converted to speech which isplayed on the telephone to the user calling in for messages.

Still another example is sending a fax message to a PDA. Fax messagesare pre-filtered based on sender and priority. The fax messages whichpass through the filter are converted to text with OCR (opticalcharacter recognition) software. The text is summarized. Data isselected using a user defined template. The text message is sent tot hePDA and the user is “notified”.

In general, a user can define a “morphing process” for messages in thecontext of any particular target device such as a pager or a cell phonewith a limited display.

The morphing process is a combination of message filtering, messagerestructuring, data conversion, data summarization, data selection andnotification steps that are configured to handle particular media typesfor particular target devices. Each user may define a set of rules andparameters for each device type defining how messages are morphed.

For example, a user may have a Voice Message-to-Pager morph definitionthat would do the following:

-   (a) filter messages based on sender and priority, removing from    further processing (i.e. leaving on the server) messages that are    not deemed urgent enough to disturb the user while out of the    office;-   (b) perform speech-to-text conversion;-   (c) summarize the text based on criteria defined by the user;-   (d) perform further filtering based on the summarized/converted    text;-   (e) organize the text in a template; and-   (f) send the message to the pager.

In general, a morphing process will include these steps in some orderdetermined by the user. In addition, message restructuring steps allowthe user to handle multiple attachments of varying media attached to themessage. For example, the user may select that a summary of theattachments be created (attachment name and media type) or may requestthat the attachments be expanded, converted and summarized as describedabove for the single media message.

There have been described and illustrated herein methods and apparatusfor processing multimedia messages. While particular embodiments of theinvention have been described, it is not intended that the invention belimited thereto, as it is intended that the invention be as broad inscope as the art will allow and that the specification be read likewise.It will therefore be appreciated by those skilled in the art that yetother modifications could be made to the provided invention withoutdeviating from its spirit and scope as so claimed.

1. A multimedia mailbox system, comprising: a message store for storingmultimedia messages; conversion means for converting messages in onemedium to messages in another medium, said conversion means convertingmessages based on a type of target device being used to access thestored message; summarization means for automatically summarizing thestored message, wherein said summarization means reduces messages to alist of keywords from a plurality of lists of keywords, each keyword ofeach list being selectable by a user, said list of keywords applied tothe stored message being based upon a sender of a message; andorganization means for organizing content of the stored message into atemplate, said template including information fields selected by a userand a number of characters for each field selected by said user, whereinsaid stored message is available for access in accordance with saidtemplate for said user.
 2. A system according to claim 1 wherein saidconversion means includes at least two selected from the groupconsisting of a text to speech converter, a speech to text converter,and a fax to text converter.
 3. A system according to claim 1 furthercomprising means for determining the type of target device being used toaccess the stored message.
 4. A mailbox system, comprising: a messagestore for storing messages; means for summarizing message content of astored message that operates as a function of a type of target devicebeing used to access the stored message, wherein said summarizationmeans reduces messages to a list of keywords from a plurality of listsof keywords, each keyword of each list being selectable by a user, saidlist of keywords applied to the stored message being based upon a senderof a message; and organization means for organizing the message contentof the stored message into a template, said template includinginformation fields selected by a user and a number of characters foreach field selected by said user, wherein said stored message isavailable for access in accordance with said template for said user. 5.A system according to claim 4 wherein said means for summarizing isoperative to summarize a stored message based on a predefined set oftarget device related parameters.
 6. A system according to claim 5wherein said predefined set of target device related parameters includestarget device display size.
 7. A system according to claim 5 whereinsaid predefined set of target device related parameters includes targetdevice buffer size.
 8. A system as set forth in claim 4 wherein saidtarget device is a mobile device.
 9. A system as set forth in claim 8wherein said mobile device is a pager.
 10. A system as set forth inclaim 8 wherein said mobile device is a PDA.
 11. A system as set forthin claim 8 wherein said mobile device is a cell phone.
 12. A method formanaging a multimedia mailbox system, comprising the steps of: storingmultimedia messages in a mailbox; converting messages in one medium tomessages in another medium, said conversion of messages based upon atype of target device being used to access the stored message;summarizing the stored message, wherein summarizing the stored messagereduces messages to a list of keywords from a plurality of lists ofkeywords, each keyword of each list being selectable by a user, saidlist of keywords applied to said text being based upon a sender of amessage; and organizing content of the stored message into a template,said template including information fields selected by a user and anumber of characters for each field selected by said user, wherein saidmessage is available for access in accordance with said template forsaid user.
 13. A method according to claim 12 wherein said step ofconverting messages includes utilizing at least two selected from thegroup consisting of a text to speech converter, a speech to textconverter, and a fax to text converter.
 14. A method according to claim12 further comprising determining a type of target device being used toaccess the stored message.
 15. A method according to claim 12 furthercomprising the step of summarizing message contents based on a type oftarget device being used to access the stored message.
 16. A method formanaging a mailbox system, comprising the steps of: a) storing messagesin a mailbox; b) summarizing message content of a stored message as afunction of a type of target device being used to access the storedmessage, wherein summarizing the stored message reduces messages to alist of keywords from a plurality of lists of keywords, each keyword ofeach list being selectable by a user, said list of keywords applied tosaid text being based upon a sender of a message; and c) organizing themessage content of the stored message into a template, said templateincluding information fields selected by a user and a number ofcharacters for each field selected by said user, wherein said storedmessage is available for access in accordance with said template forsaid user.
 17. A method according to claim 16 wherein said step ofsummarizing is operative to summarize a stored message based on apredefined set of target device related parameters.
 18. A methodaccording to claim 17 wherein said predefined set of target devicerelated parameters includes target device display size.
 19. A methodaccording to claim 17 wherein said predefined set of target devicerelated parameters includes target device buffer size.
 20. A systemaccording to claim 1 wherein information fields include sender name,time, summary, message priority and un-summarized text.
 21. A systemaccording to claim 4 wherein information fields include sender name,time, summary, message priority and un-summarized text.
 22. A methodaccording to claim 12 wherein information fields include sender name,time, summary, message priority and un-summarized text.
 23. A methodaccording to claim 16 wherein information fields include sender name,time, summary, message priority and un-summarized text.
 24. A multimediamailbox system according to claim 1 wherein said list of keywordsapplied to said text being further based upon message subject
 25. Amultimedia mailbox system according to claim 1 further comprising:filter means for determining high priority messages; and automatic meansfor automatically sending a summary of high priority messages to adesignated user.
 26. A method according to claim 12 further comprisingthe steps of: determining which messages have high priority; andautomatically sending a summary of high priority messages to adesignated user.
 27. A method according to claim 26 wherein said step ofdetermining further comprises the step of filtering to determine messagepriority.